Composing Bosh Manifests with Spiff

Composing BOSH manifests with Spiff Dr Nic Williams

CEO of Stark & Wayne, a consultancy for people who love Cloud Foundry, BOSH and DevOps in general.

The declarative nature of a BOSH manifest can give a team solid confidence about what is actually running in production. Give BOSH a deployment manifest and BOSH will make it happen.

But if the manifest is 3000 lines long, and you have three of them - sandbox, staging and production - your solid confidence can evapourate.

"Exactly what is running on production and what is on staging?"

Spiff Spiff is a CLI tool that lets you compose smaller YAML templates together into the final large BOSH deployment manifest.

One way to install spiff is via the latest traveling-bosh installer release.

curl -k -s https://raw.githubusercontent.com/cloudfoundry-community/bosh_cli_install/master/binscripts/installer | bash Comprehension The benefit comes from authoring your spiff templates to be easily comprehensible from the title of the template.

For example, these templates for a Cloud Foundry deployment immediately indicate what concept they are describing to a team:

tiny/cf-tiny-dev.yml cf/cf-infrastructure-aws.yml tiny/cf-blobstore-nfs.yml tiny/cf-lb-haproxy.yml cf-no-ssl.yml cf-simple-secrets.yml cf-networking.yml At a glance we can see: It's a tiny deployment of Cloud Foundry, targeting AWS, using NFS for the blobstore, and HAProxy for the load balancer, but without any SSL certificates.

That's one reason you want to use spiff. Comprehension.

Modularity From the list of templates above you can immediately imagine interchanging them for alternate configuration.

cf-no-ssl.yml could change with cf-self-sert-ssl.yml or cf-ssl.yml tiny/cf-blobstore-nfs.yml could change with tiny/cf-blobstore-s3.yml to use AWS S3 for blobstores Spiff templates can provide modularity, which can allow you to compose your final BOSH deployments differently without much fuss.

Spiff Merge To compose templates together you use the spiff merge subcommand.

spiff merge \ tiny/cf-tiny-dev.yml \ cf/cf-infrastructure-aws.yml \ tiny/cf-blobstore-nfs.yml \ tiny/cf-lb-haproxy.yml \ cf-no-ssl.yml \ cf-simple-secrets.yml \ cf-networking.yml \ production-stub.yml This would output a full BOSH manifest.

All the YAML files are templates, except the last file. This is the "stub".

The stub allows you to override aspects of the templates; and to provide missing values. If you forget to provide a missing value, spiff merge will fail.

Pipelines The stub allows you to reuse the same templates for multiple deployments - such as your CI pipeline - but with a few deployment-specific changes.

For example: For a sandbox deployment, to different AWS VPC networking and perhaps with non-production sizing, you would have a stub sandbox-stub.yml.

How to write templates Some BOSH releases - such as Cloud Foundry's cf-release include a set of templates: https://github.com/cloudfoundry/cf-release/tree/master/templates. But you might find they are not modular or comprehesible enough for your team.

Other BOSH releases may not include any spiff templates at all.

In either case you will want to know how to write spiff templates to suit your team. Don't feel locked into any upstream templates.

A spiff template and a spiff stub are just a YAML files.

Given a stub sandbox-stub.yml:

name: sandbox-cf director_uuid: 21a7a7ee-d7f1-11e4-9dd5 compilation: cloud_properties: instance_type: c1.medium And a template cf-deployment.yml:

name: my-cf director_uuid: (( merge )) compilation: workers: 6 cloud_properties: (( merge )) The result is:

$ spiff merge cf-deployment.yml sandbox-stub.yml compilation: cloud_properties: instance_type: c1.medium workers: 6 director_uuid: 21a7a7ee-d7f1-11e4-9dd5 name: sandbox-cf What happened?

The (( merge )) token was replaced by the director_uuid value from the stub. The key name was overridden by the stub's own value sandbox-cf The key compilation and its subkey compilation.workers was unaffected and included in the final YAML The keys are all reordered alphabetically. Don't know why it does that. I liked my ordering. Merge errors If our stub did not provide overrides for all (( merge )), then spiff merge will fail.

If our stub is only the name:

$ spiff merge cf-deployment.yml sandbox-stub.yml 2015/03/31 15:13:33 error generating manifest: unresolved nodes: (( merge )) in cf-deployment.yml compilation.cloud_properties (( merge )) in cf-deployment.yml director_uuid This error tells us that the cf-deployment.yml template requires two keys to be provided by a stub; or another template: compilation.cloud_properties and director_uuid.

Multiple templates compilation.cloud_properties looks like its ripe for a modular solution. If we're on AWS then we'll use one answer; and on bosh-lite we'll use another.

Template cf-infra-aws.yml:

compilation: cloud_properties: instance_type: c1.medium Template cf-infra-bosh-lite.yml:

compilation: cloud_properties: dummy: here And we'll put director_uuid back in the per-deployment stub:

name: sandbox-cf director_uuid: 21a7a7ee-d7f1-11e4-9dd5 We can now create a manifest for bosh-lite or AWS:

$ spiff merge cf-deployment.yml cf-infra-bosh-lite.yml sandbox-stub.yml compilation: cloud_properties: dummy: here workers: 6 director_uuid: 21a7a7ee-d7f1-11e4-9dd5 name: sandbox-cf meta: convention for stub inputs As spiff was being developed and simultaneously being adopted by the cf-release project, a convention evolved for describing the inputs to your templates: top-level meta key.

This also becomes a clean way to describe defaults in templates that are used in many locations.

It's also very useful for allowing you to override an entire list.

Let's specify the list of releases to be used by a deployment in the cf-deployment.yml:

meta: releases:

releases: (( meta.releases ))

compilation: workers: 6 cloud_properties: (( merge )) The snippet (( meta.releases )) means to copy in the value from meta.releases in the same template.

You could optionally now override the entire releases array in the stub:

name: cf version: 205 name: monitor-server version: latest Examples Each of the examples above are available in a GitHub repo: git clone https://github.com/cloudfoundry-community/using-spiff-examples.git cd using-spiff-examples ./01-simple-merge/make_manifest.sh ./02-missing-fields/make_manifest.sh ./03-modular-infra/make_manifest.sh bosh-lite ./04-meta-releases/make_manifest.sh aws More usage The README for spiff merge includes many other options for creating composable, modular spiff templates under Usage.

Dr Nic Williams WRITTEN BY

Dr Nic Williams CEO of Stark & Wayne, a consultancy for people who love Cloud Foundry, BOSH and DevOps in general. Palo Alto, CA, USA http://drnicwilliams.com Published on March 31, 2015 SPREAD THE WORD